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Abstract 

Practitioners and policymakers rely on meta-analyses to inform decision-making around the 
allocation of resources to individuals and organizations. It is therefore paramount to consider the 
validity of these results. A well-documented threat to the validity of research synthesis results is 
the presence of publication bias, a phenomenon where studies with large and/or statistically 
significant effects, relative to studies with small or null effects, are more likely to be published 
(Rothstein, Borenstein, & Sutton, 2005). We investigated this phenomenon empirically by 
reviewing meta-analyses published in top tier journals between 1986-2013 that quantified the 
difference between effect sizes from published and unpublished research. We reviewed 383 
meta-analyses of which 81 had sufficient information to calculate an effect size. Results 
indicated that published studies yielded larger effect sizes than those from unpublished studies ( d 
= 0.18, 95% Cl [0.10, 0.25]). Moderator analyses revealed that the difference was larger in 
meta-analyses that included a wide range of unpublished literature. We conclude that 
intervention researchers require continued support to publish null findings and that meta- 
analyses should include unpublished studies to mitigate the potential bias from publication 
status. 


Keywords: publication bias, meta-review, effect size, meta-analysis. 
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Estimating the Difference between Published and Unpublished Effect Sizes: A Meta- 

Review 

Practitioners and policymakers rely on high quality primary research as well as meta- 
analyses to inform decision-making around the allocation of resources and provision of services 
to individuals and organizations. This paradigm extends to the results of studies both null and 
significant because ensuring the discontinuation of ineffective, or worse, harmful programs, is as 
important as ensuring effective programs are distributed widely. Greenwald (1975), in a 
landmark study on the “prejudice against the null hypothesis” (p. 1), posited that null research 
findings are often interpreted as a failing of the researcher (or design), instead of the probable 
hypothesis that successfully detecting a significant effect is much less likely. Numerous studies 
have shown, however, that across the peer-reviewed, published literature detecting a statistically 
significant effect is often the rule rather than exception (Ionnidis, 2005; Ioannidis & Trikalinos, 
2007). As a result, there have been calls recently by journals and researchers alike to include 
null findings from high quality studies (Cumming, 2013). 

Despite these well-intentioned calls to action, statistically significant research remains the 
norm in peer-reviewed, published research (Rothstein, Borenstein, & Sutton, 2005). Indeed, 
prior research has documented that (a) studies with non-significant findings are often left 
unpublished (Ferguson & Heene, 2012), (b) authors have a tendency to report results for 
outcomes with statistically significant effects (Pigott, Valentine, Polanin, Williams, & Canada, 
2013), and (c) external referees tend to provide more favorable peer reviews to studies with large 
or statistically significant effects (Lee, Sugimoto, Zhang, & Cronin, 2013). In addition, 
favorable outcomes receive quicker time to publication (Ioannidis, 1998) and a higher rate of 
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citation (Tanner-Smith & Polanin, 2014). The totality of these biases is often referred to as 
dissemination bias. 

The proliferation of dissemination biases has the potential to impact not only the validity 
of meta-analytic results, but those of primary research as well. Dissemination biases create the 
illusion of theory confirmation, potentially leading to the continuation of programs or policies 
that are ineffective or, worse, harmful. Moreover, continuation of funds to ineffective programs 
inhibits the growth of potentially new and important research. At a minimum, dissemination bias 
in a particular field may cause an overestimation of potential future effects, which has an impact 
on researchers’ preliminary power and cost analyses (Rothstein et al., 2005). The recent 
movement to represent clinical trials or funded research in data repositories, such as 
ClinicalTrials.gov, addresses this concern by mandating that primary researchers publish a 
study’s results, regardless of the study’s findings. 

Erroneous conclusions or misleading results reporting are particularly problematic for 
meta- analyses that synthesize evidence on the effectiveness of educational or social 
interventions. Fortunately, most authors of research syntheses are aware of the dangers of 
publication bias (Ferguson & Brannick, 2012), and there are several analytic methods available 
for meta-analysis authors to assess the possibility and potential impact of publication bias on 
their review findings (Sutton, 2009). One common method is to test for differences in mean 
effect sizes as a function of publication status — that is, asking whether the observed effects are 
larger or smaller in published versus unpublished studies included in the meta-analysis (Lipsey, 
2009). This empirical test addresses whether excluding unpublished literature would have 
changed the meta-analytic findings, and is generally used to elucidate the potential dangers of 
having only including published literature in the meta-analysis. 
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The purpose of this meta-review is to capitalize on these tests of average effect size 
differences by publication status. Our primary research question is: What is the observed 
difference in mean effect sizes for published versus unpublished studies, as reported in meta- 
analyses in the fields of education and psychology? The project’s results provide an estimate of 
possible dissemination bias, and, moreover, have the potential to inform both primary 
researchers and meta-analysts of the phenomenon of dissemination bias within the social science 
literature. We chose to focus this meta-review on education and psychology because less 
attention has been paid to publication bias in the social sciences relative to health and medical 
sciences (Rothstein, et al., 2005). 

Research on Publication Bias 

Systematic literature reviews aim to locate and include all relevant studies on a particular 
topic using transparent procedures designed to maximize replicability and minimize potential 
biases. The inclusion of all relevant studies that meet pre-specified eligibility criteria means that 
studies should be included in a review regardless of publication status, and thus most systematic 
reviews attempt to include studies controlled by academic/commercial publishers as well as grey 
literature. Grey literature refers broadly to any study “which is produced on all levels of 
government, academic, business, and industry in print and electronic formats, but which is not 
controlled by commercial publishers” (Hopewell, Clarke, & Mallett, 2005, p. 49). Henceforth, 
we refer to any study report outside of commercially published journal articles as unpublished 
grey literature. Dissertations and theses are one of the most common type of grey literature 
included in systematic reviews, but other forms of grey literature may include conference 
abstracts/presentations, books and book chapters, unpublished technical reports or white papers, 
and unpublished datasets received from primary study authors. 
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Dissemination and publication biases. The primary reason meta-analysts deliberate and 
advocate for the inclusion of grey literature in systematic reviews is publication bias, or the 
higher likelihood for studies with large or statistically significant findings to be published 
relative to studies with small or null effects (Rothstein et al., 2005). Publication bias is one form 
of bias under the larger umbrella of dissemination bias, which refers to the phenomenon whereby 
the selective reporting, selective publication, and/or selective inclusion of scientific evidence in a 
systematic review yields a biased answer, on average (Bax & Moons, 2011). Other 
dissemination biases, for instance, might include time-lag bias (studies with large/significant 
results are published more quickly), citation bias (studies with large/significant results are cited 
more frequently), and outcome -reporting bias (outcomes with large/significant results are more 
likely to be reported by primary study authors). 

Numerous empirical studies have documented the existence of dissemination biases, 
mostly in the medical sciences. For instance, Easterbrook, Berlin, Gopalan, and Matthews 
(1991) surveyed projects approved by the Central Oxford Research ethics committee in the 
1980s, and found that studies with statistically significant findings were more likely to be 
published. Similarly, Decullier, Lheritier, and Chapuis (2005) retrospectively investigated 
studies approved by French research ethics committees in the 1990s and found that studies with 
significant and positive results were more likely to be published relative to those with 
inconclusive or non-confirmatory findings. Chan, Hrobj arts son, Haahr, Gptzsche, and Altman 
(2004) reviewed protocols of randomized controlled trials conducted in Denmark and found that 
whereas 71% of statistically significant outcomes were reported, only 56% of non-significant 


outcomes were reported. 
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Reviews of medical meta-analyses have yielded similar findings. Hopewell, Loudon, 
Clarke, Oxman, and Dickersin (2009) reviewed Cochrane reviews to investigate the association 
between publication status and research findings for registered clinical trials and found that trials 
with statistically significant results were more likely to be published. Dwan and colleagues 
(2008) examined systematic reviews of medical interventions and found that 40-62% of studies 
“had at least one primary outcome that was changed, introduced, or omitted” (p. 1). A recent 
review of 51 meta-analyses across the communication sciences literature (Levine, Kelli, & 
Carpenter, 2009) showed that effect sizes were negatively correlated with sample sizes. Their 
findings indicated that statistically significant findings were more likely to be published 
especially among smaller sample studies (instead of small sample studies published regardless of 
effect), and the authors concluded that resulting meta-analyses would likely overestimate the true 
population effect. Thus, the medical and communication sciences literature has a well- 
documented trend toward publishing statistically significant and large treatment effects 
(Rothstein et al., 2005). 

Despite consistent evidence of publication bias in the medical literature, Torgersen 
(2006) asserted that few studies have examined its presence in the educational or psychological 
sciences. Of the studies conducted, for example, Smith’s (1980) review of 12 meta-analyses 
from education found that, on average, studies published in peer-reviewed journals had a mean 
effect size one-third of a standard deviation larger than unpublished studies. Kulik and Kulik 
(1989) found similar results when examining their own four meta-analyses in computer-based 
instruction in elementary and secondary levels, ability grouping, and mastery learning systems. 
The authors compared the effect sizes between published and unpublished literature 
(dissertations) and found evidence of publication bias in two of the four meta-analyses. They 
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concluded that publication bias was only one possible factor, however, because dissertation 
authors were less experienced researchers and this could explain the lack of publication. Lipsey 
and Wilson (1993) also examined publication bias in education by reviewing 302 meta-analyses 
of education-based studies. Of the 302 meta- analyses reviewed, only six reported effect sizes 
that were negative and more than 85% of studies reported effect sizes larger than 0.2 (in the 
standardized mean difference metric). More recently, Pigott et al. (2013) investigated outcome- 
reporting bias in education by tracking dissertations after they were later published in journals. 

Of the 79 dissertation studies that were subsequently matched with a later journal publication, 36 
appeared to experience some form of outcome censoring due to statistical significance. The non- 
statistically significant outcomes were 30% less likely to appear in publication compared to 
statistically significant outcomes. Clearly, publication bias can have an impact on the validity of 
meta-analyses and their conclusions. 

Methods for assessing potential publication bias. Several analytic methods have been 
proposed to assess the potential presence and impact of publication bias in a meta-analysis 
(Ferguson & Brannick, 2012; Ioannidis, Munafo, Fusar-Poli, Nosek, & David, 2014). These 
techniques vary in complexity (Hedges & Vevea, 2005) and rigor (Rosenthal, 1991), although 
some techniques tend to be more popular with review authors (Ahn, Ames, & Myers, 2012). 

One visual analysis technique for assessing the possibility of publication bias in a meta-analysis 
is an inspection of a funnel plot (Sterne, Becker, & Egger, 2005). Funnel plots are scatter plots 
of effect size estimates (typically shown on the x-axis) and some measure of precision of the 
effect size estimate (typically the inverse of the standard error or sample size, shown on the y- 
axis). In the absence of publication bias, one expects the effect sizes to be dispersed evenly 
throughout the graph and to be symmetric around the mean effect size. If studies with smaller 
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sample sizes and small/null/negative effect sizes are not included in the meta-analysis, the funnel 
plot appears either asymmetric or the bottom half plot empty. There are numerous problems 
with this visual method for assessing potential publication bias; foremost of which is that it relies 
on a subjective visual assessment of the a/symmetry of the plot, or that plot asymmetry could be 
due to numerous factors other than publication bias (Schild & Voracek, 2014). 

Due to the limitations of visually inspecting funnel plots, other methods have been 
developed to attempt to quantify the asymmetry in the funnel plot. For example, regression- 
based approaches attempt to estimate the magnitude of association between effect size estimates 
and their precision and include such tests as Egger, Peters, and Harbord tests for funnel plot 
asymmetry (e.g., Egger, Smith, Schneider, & Minder, 1997; Harbord, Egger, & Steme, 2006; 
Peters, Sutton, Jones, Abrams, & Rushton, 2006; Rucker, Schwarzer, & Carpenter, 2008). 
Alternatively, the trim and fill method (Duval & Tweedie, 2000) is another commonly used 
method that estimates the number of missing studies in a meta-analysis, assuming a symmetric 
funnel plot should exist. The procedure does so by assessing funnel plot asymmetry and then 
deducing the number of potentially missing studies by attempting to produce a symmetric funnel 
plot. As a result, one may calculate an average effect size that includes imputed missing effect 
sizes and compare this value to the original effect size generated without imputed values. 

Rosenthal (1991) and Orwin (1983) proposed various measures of “file-drawer” or 
“failsafe N” statistics, which estimate the number of presumably missing studies that could be 
added to a meta-analysis before it yielded a null mean effect size. This statistic has recently 
fallen out of favor with methodologists because it can be easily misrepresented and null findings 
may not always produce effect sizes close to zero (Becker, 2005; Higgins & Green, 201 1). The 
p-curve is another method that attempts to uncover selective reporting, or “p-hacking,” in 
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primary reports (Simonsohn, Nelson, Leif, & Simmons, 2013). Although all of the 
aforementioned techniques are useful for assessing the possibility of publication bias in a meta- 
analysis, they typically do not focus on the observed difference in mean effect sizes for published 
and unpublished studies’ effect sizes. 

As noted previously, one of the most common methods for assessing the possibility of 
publication bias in a meta-analysis is to test for the differences in mean effect sizes across the 
published versus unpublished studies included in the review (Polanin & Pigott, 2014). This test 
is commonly conducted as a simple moderator analysis to examine whether the observed 
difference in mean effect sizes across the published and unpublished studies is larger than would 
be expected due to chance. When comparing means across two groups (i.e., published vs. 
unpublished studies), this moderator test can be converted to an effect size. Consideration of 
more than two groups, for example, when the purpose is to test for differences between multiple 
types of grey literature (e.g., journal articles, books, dissertations), results in the authors utilizing 
a (Ubctwccn test (analogous to an F-test in an ANOVA model). Either the result of the Q- 
between or the z-score test provides a measure of the magnitude of the observed difference in 
mean effect sizes across the publication status groups. 

The Current Study 

This meta-review synthesizes results across meta-analyses in the fields of education and 
psychology, in an attempt to capture the average magnitude of difference in mean effect sizes 
across published and unpublished studies. The purpose of this meta-review is to capitalize on the 
moderator tests of differences in mean effect sizes by publication status. Our primary research 
question is as follows: What is the observed difference in mean effect sizes for published versus 
unpublished studies, as reported in meta-analyses in the fields of education and psychology? 
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Our secondary research question is whether any characteristics of the meta-analyses themselves 
may influence the magnitude of observed differences in mean effect sizes between published and 
unpublished studies. 

Method 

Search and Screen 

The purpose of this meta-review is to estimate the average difference in the magnitude of 
effect sizes between published and unpublished primary studies, as reported in existing meta- 
analyses in the fields of education and psychology. As such, we screened all published meta- 
analyses in the journals Review of Educational Research and Psychological Bulletin. These two 
journals were chosen to represent education and psychology because they publish high-quality 
meta-analyses and because they each consistently generate large impact factors. To be eligible 
for inclusion in this meta-review, the published research synthesis must have been conducted 
between 1986 - 2013 and must have presented quantitative synthesis results (i.e., conducted a 
meta-analysis). The nature of the meta-review also meant that the meta-analyses must have 
included unpublished, grey literature. There were no restrictions placed on the type of studies 
synthesized (e.g., meta-analyses could synthesize experimental intervention effectiveness 
research or non-experimental correlational research) or the method of synthesis. We were 
interested in producing a generalizable estimate of the difference in effect sizes and therefore 
purposively enabled the inclusion criteria to be broad. 

Each citation’s title and abstract was hand screened for inclusion. If a study appeared to 
meet these broad screening criteria, we retrieved the full-text version and screened the full article 
for final eligibility determination. Across the two journals, we screened a total of 1,858 
citations; 383 citations appeared to meet the inclusion criteria and were retrieved for full-text 
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screening. One researcher screened all 383 articles and another researcher randomly selected 
and screened 10% of the studies. The two reviewers agreed on the decision to include the meta- 
analysis often (96%) and the disagreements were resolved via consensus. Of the screened 383 
articles, 81 met the inclusion criteria and had sufficient information available to calculate an 
effect size. 

Coding 

A comprehensive coding manual was created for this project and four major sections 
guided the coding process. The first section coded basic information about the meta- analysis: 
Author names, date of publication, the title of the article, the publication source, and whether the 
meta- analysis was funded. The second section detailed the meta-analyses’ characteristics: A 
description of the synthesis, outcome construct type (educational achievement or psychological), 
effect size metric (standardized mean difference or correlation coefficient), and type of synthesis 
model (fixed-effect, random-effects, or both). The third major section detailed the types of grey 
literature included in each meta- analysis. On a few occasions, the meta-analysis authors failed to 
delineate the types of grey literature provided or simply stated that “unpublished studies” were 
included in the review. In these cases, we labelled this type of grey literature as “general.” We 
also coded the total number of published and unpublished studies included in each meta-analysis. 
The fourth section of the coding manual detailed information needed to extract effect sizes from 
each meta- analysis. 

Effect Size Extraction 

The effect size of interest was the difference between the average effect sizes for 
published versus unpublished literature included in each meta-analysis. This can be represented 
as 
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Diffi — T Published Tunpublished EQ1 

where T Published represented the average effect size from published literature and T Unpu bUshed 
represented the average effect size from unpublished literature in any given meta-analysis. This 
effect size was calculated such that positive values indicated published studies had larger effects 
than unpublished studies. The standard error of this effect size is the sum of the standard errors 
for each average effect size. If the meta-analyses presented effect sizes for studies across 
multiple types of unpublished literature categories (e.g., dissertations, conference abstracts, etc.), 
we first combined the multiple sources into one category (i.e., all unpublished literature) before 
estimating the above effect size statistic. 

Because meta-analyses reported the results of this moderator test in a variety of forms, 
we could not always directly calculate this simple effect size statistic. For example, a portion of 
the meta-analyses provided only the (Ubetween test statistic with 1 df. In this case, we converted 
the (Ubctwcen statistic with 1 df to the z-metric by taking its square root. As such, and in order 
to synthesize effects across all meta-analyses included in the sample, we transformed all effect 
sizes into the z-metric following Borenstein, Hedges, Higgins, and Rothstein (2009). For each 
meta-analysis i that provided information to calculate a difference estimate, we calculated a z- 
statistic: 

_ Diffi EQ2 

Zi SEi 

where Diffi is the difference calculated in equation (1) and the SE, is the standard error for meta- 
analysis i. Again, this z-statistic effect size was coded such that positive values indicated 
published studies had larger mean effects than unpublished studies. 

Finally, to guard against potential outcome reporting biases by review authors, we 
estimated the difference between published and unpublished studies directly from included 
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summary tables when the authors reported that they conducted the moderator test but failed to 
provide empirical results. For instance, Elfenbein and Ambady (2002) indicated that a 
moderator test was conducted, but that the test yielded a non-significant result. The authors, in 
addition, provided a table of every study included in the meta-analysis as well as the effect size 
extracted. As a result, we extracted this summary table and directly calculated the effect size 
difference of interest. 

Analysis 

We conducted a random-effects meta-analysis, using robust variance estimation, to 
synthesize the weighted z-stati sties estimated from each meta-analysis (Hedges, Tipton, & 
Johnson, 2010; Tanner-Smith & Tipton, 2014). The use of robust standard errors allows for the 
inclusion of multiple effect sizes per study. Similar to traditional meta-analysis, the weight of 
each study remained the inverse of the z-statistics’ variance (i.e., the square of the standard error; 
Lipsey & Wilson, 2001). The result is an inverse-weighted, average z statistic. The standard 
error of the weighted average z, used to calculate confidence intervals, is the square root of the 
inverse of the sum of the weights. Congruent with standard meta-analytic procedures, we also 
estimated the heterogeneity among the meta-analyses via the x and 7" statistics (Higgins & 
Thompson, 2002). To answer our first research question and provide an estimate of the 
magnitude of the difference between published and unpublished studies, we transformed the 
weighted average z back into the standardized mean-difference metric (Borenstein et al., 2009). 

To answer our second research question, we also conducted moderator tests to determine 
if the effect sizes varied as a function of different subgroups of meta-analyses. Given the 
exploratory nature of these analyses, we chose to conduct these tests using a meta-regression 
framework, first by conducting univariate meta-regression models and then conducting a 
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multivariable meta-regression model. We explored the following potential effect size 
moderators: effect size metric (standardized mean difference or correlation coefficient), outcome 
construct type (achievement or psychological outcome), whether the meta-analysis included 
dissertations as their only grey literature source, whether the estimated effect size derived from a 
(7-bctween model, whether the review received funding, the percentage of unpublished literature 
included in the meta-analysis, and the date of publication. Model fit was assessed using an 
adjusted R 2 (Cheung, 2013). All calculations and figures were prepared using the R package 
robumeta (Fisher & Tipton, 2014). 

Results 

Descriptive Results 

This meta-review synthesized effect sizes from 81 meta-analyses, 27 meta-analyses 
derived from the Review of Educational Research and 54 from Psychological Bulletin (see Table 
1). A majority of the meta-analyses used the standardized mean-difference effect size metric ( k 
= 57, 70.4%) and the rest used the correlation coefficient ( k = 24, 29.6%). Most effect sizes 
indexed effects for psychological outcomes ( k = 47, 58.0%); others measured achievement 
outcomes ( k = 30, 37.0%) and a few reported both types of outcomes ( k = 4, 4.9%). About one- 
half of the meta-analyses reported that they received some type of funding ( k = 44, 54.3%). A 
large percentage of meta-analyses failed to report the type of synthesis model utilized ( k = 25, 
30.9%) or used a random-effects analysis (k = 35, 43.2%); a few reported using a fixed-effect 
model ( k = 10, 12.3%) or both fixed- and random-effects models (k = 11, 13.6%). Seven meta- 
analyses (8.6%) indicated that a moderator test was conducted and failed to provide relevant 
summary statistics, but included a summary table of the studies and effect sizes which could be 
used to estimate a difference statistic. Most of the meta-analyses were published after 2000 ( k = 
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58, 71.6%). Figure 1 provides a descriptive graph of the number of meta-analyses published 
each year and includes how many of these meta-analyses reported conducting publication bias 
moderator analyses. 

It is also informative to describe the quantity and types of grey literature included (Table 
2). We should note, however, that to be included in this review, the meta-analyses must have 
included grey literature and conducted a test of the difference in effect size magnitude by 
publication status. Therefore, the percentages of included grey literature may not reflect the 
population of systematic reviews because not all systematic reviews include grey literature 
(Polanin & Pigott, 2014). For this sample of meta-analyses, 68.5% of the included primary 
studies were journal articles (Mean number of studies = 55.74, SD = 56.74) and 31.5% were 
some form of grey literature ( M = 22. 19, SD = 23. 1 1). The most common type of grey literature 
was dissertation abstracts ( M = 13.43, SD = 18.42). The next most commonly used grey 
literature type was general unpublished literature (M = 5.03, SD = 2.96); this category included 
unpublished datasets, government reports, documents retrieved from database searches but not 
published elsewhere, or a mix of unpublished articles where the authors did not distinguish the 
study type. Review authors included conference abstracts (M = 2.34, SD = 8.20), books ( M = 
0.56, SD = 2.94), and book chapters (M = 0.82, SD = 3.40) infrequently. Across all meta- 
analyses, 31.7% included dissertations/theses (hereafter referred to as dissertations) as their only 
source of grey literature. 

Analytical Results 

The 81 meta-analyses provided information to calculate 147 "-statistics that indexed the 
difference in mean effect sizes for published versus unpublished studies (Figure SI). The 
average study contributed more than one effect size (M = 1.80, SD = 2.05, Min = 1, Max = 14). 
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Synthesizing the z-statistics across all 147 effect sizes, using robust standard errors to account for 
within-study clustering, yielded a statistically significant average effect (z = 0.80, 95% Cl [0.45, 
1.15]), indicating that published studies had significantly larger mean effects than unpublished 
studies. We transformed the inverse-weighted, average z-statistic into the d-metric (d = 0.18, 
95% Cl [0.10, 0.25]). This indicates that on average, published studies reported effects that were 
0.18 standard deviations larger than the effects reported in unpublished studies. 

We observed a large amount of heterogeneity among the effect sizes (x = 5.86 , 1 = 
99.8%). Therefore, we also conducted exploratory meta-regression analyses to examine whether 
the difference in mean effects between published and unpublished studies varied according to 
characteristics of the meta-analyses (Table 3). Only one of the six meta-regression models 
yielded traditional statistically significant differences (i ,e.,p < .05). Namely, meta-analyses that 
included dissertations as their only source of grey literature yielded a statistically significantly ( b 
= -0.77, 95% Cl [-1.51, -0.03]) lower average z-statistic (z = 0.28, 95% Cl [-0.33, 0.90]) relative 
to studies that included all types of grey literature (z = 1.03, 95% Cl [0.60, 1.45]). In other 
words, meta-analyses that only included dissertations as their grey literature source concluded 
that the difference between published and unpublished studies’ (i.e., dissertations) effect sizes 
were not substantial. To account for covariance among the variables and further ensure the 
validity of these results, we also conducted a multiple predictor meta-regression model that 
included all of the hypothesized variables. The results again indicated that the only significant 
result derived from dissertations being used as the only source of grey literature ( b = -0.83, 95% 
Cl [-1.63,-0.03]). 

Although no other meta-regression analyses yielded significant results, we believe it is 
important to highlight a few other results as potential areas for further research. For instance, 
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results indicated that ( b = 0.27, 95% Cl [-0.42, 0.95]) meta-analyses receiving funding had a 
larger difference (z = 0.92, 95% Cl [0.35, 1.49]) relative to unfunded studies (z = 0.65, 95% Cl 
[0.26, 1.05]). One possible explanation for this is the amount and types of unpublished literature 
included in funded meta-analyses. Funded meta-analyses included more primary studies 
(Funded: M = 77.00, SD = 70.65; Unfunded: M = 70.50, SD = 52.85), and 32 of the 44 funded 
studies (72.7%) included diverse sources of grey literature (i.e., sources other than dissertations) 
whereas only 25 of the 38 unfunded meta-analyses (65.7%) included grey literature sources other 
than dissertations. Although this analysis was purely exploratory, the trend indicates that funded 
meta-analyses may include more diverse types of grey literature and thus yield a more 
conservative, yet potentially less biased, estimate of the overall effect size. 

Discussion 

The purpose of this meta-review was to summarize the observed differences in effect 
sizes between published and unpublished literature, as reported in meta-analyses in the 
educational and psychological sciences. After a comprehensive search, screen, and selection 
process of the citations from two leading review journals, we identified 81 meta-analyses and 
147 effect sizes that were eligible and included in this meta-review. We coded and extracted 
effect sizes that represented the magnitude of difference in the mean effect sizes for published 
and unpublished studies included in the meta-analyses. Results from the meta-review indicated 
that, on average, published studies had mean effect sizes that were 0.18 standard deviations 
larger than the mean effect sizes for unpublished studies included in the meta- analyses (95% Cl 
= 0.10, 0.25). Exploratory moderator analyses indicated that meta-analyses that included 
dissertations as their only grey literature source were significantly less likely to observe effect 
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size differences between published and unpublished studies. Overall, these results provided 
evidence of the existence of publication bias in education and psychology. 

Limitations 

Several limitations should be considered. For instance, this meta-review included studies 
from two review journals only. Although we believe these journals represent the utmost quality 
and superior methodology, it is still a limited sampling frame. It is possible that the meta- 
analyses included in these journals represent only one type of review and therefore 
systematically differ from the population of reviews. We posit, however, that because these 
journals are some of the most selective in the fields, the problem of publication bias might well 
be under-e stimated: The meta-analyses included in these selective journals may include more 
grey literature than those published in less selective or competitive journals. Future research 
should attempt to replicate these findings using meta-analyses published across multiple journals 
in education and psychology. 

It is also important to consider that in the current study, the effect size estimates of the 
difference between published and unpublished studies represent averages of averages. Indeed, 
the totality of primary studies represented in the meta-review is large ( k = 6,392). It is entirely 
possible, in fact plausible, that a number of these effect sizes will have inherent bias due to any 
number of factors. Averaging these effect sizes may balance this bias and render it nominal, but 
it is nonetheless important to note that we are providing seemingly precise estimates for rather 
crude approximations. Thus, although we are confident in the overall conclusion that published 
studies yield larger average effects than unpublished studies, it is difficult to state with great 
certainty the exact magnitude of the average difference between published and unpublished 
effect sizes. As with any statistical estimate, a level of uncertainty is inherent in these estimates. 
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Finally, and most importantly, it is highly probable that the meta-analytic literature, 
similar to primary literature (Pigott et al., 2013), suffers from outcome reporting biases. In other 
words, it is likely that meta-analysts test for differences between published and unpublished 
primary studies but report the results of this test only when the test is statistically significant. We 
believe this is highly probable because review authors often make such statements. Wood, 
Lundgren, Ouellette, Busceme, and Blackstone’s (1994) moderator results, for example, 
indicated that unpublished studies actually yielded stronger effect sizes relative to published 
studies for one outcome. However, the authors conducted several syntheses and tested for 
differences between published and unpublished studies, yet only reported the moderator results 
when significant differences were found. As a result, this particular meta-analysis provided a 
somewhat biased estimate of the difference between published and unpublished studies, albeit in 
an opposite direction than we hypothesized. Nevertheless, this is one clear example of outcome 
reporting bias. Meta-analysts, in the future, would do well to follow the reporting practices set 
forth by primary researchers with regard to outcome reporting. 

Future Research 

The social sciences need greater awareness of publication bias and its potential negative 
effects, particularly given the increased role of meta-analyses in informing policy and practice 
(Ferguson & Brannick, 2012). Yet, as Rothstein et al. (2005) argued, little empirical research 
has focused on publication bias in the social sciences, relative to the medical sciences, where it 
has been researched extensively. For instance, we advocate for greater use of cohort studies in 
the social and behavioral sciences that track the research process of federally-funded primary 
studies (e.g., Chan et al., 2004). These types of studies can help elucidate where dissemination 
bias occurs and how to potentially stop it. As for meta-analysis, future research is needed on 
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how to handle the presence of publication bias. One promising option is the use of selection 
models that account for outcome-reporting bias and adjust parameters accordingly (Hedges & 
Vevea, 2005). These models are still underdeveloped, however, and offer response to only one 
type of dissemination bias. Clearly, there is a need for new innovations in meta-analysis 
methodology that can be used to assess and address the problems associated with publication 
bias. 

Implications 

Given the rise in meta-analyses and their broad reach, it should be paramount to ensure 
that the results of meta-analyses are valid. Policymakers and practitioners rely on meta-analysis 
results to guide decision-making and decide which programs receive support. Stakeholders rely 
on results from meta-analyses due to the implicit understanding that they represent the totality of 
evidence for a given research question. Although it is tempting, and often less time-consuming, 
to only include studies from peer-reviewed journals in a meta-analysis, researchers must ensure 
that all available databases, resources, and contacts have been exhausted in the literature search 
for a meta-analysis. 

Publication bias is not only a problem for meta-analysis. The over-represented 
publication of statistically significant findings has the potential to disrupt the development of 
new theories and theory testing, research replication, future study planning, and the allocation of 
resources to research. All of these issues concern the primary researcher, as well as the meta- 
analyst. Primary researchers should take pains to publish the confirmation as well as the 
contradiction of theory and program testing. 
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Conclusions 

Evidence is available from methodological reviews to support the existence of 
publication bias, particularly in the health and medical fields (Rothstein et al., 2005). The 
current study is one in a growing body of literature that suggests publication bias is also 
prevalent in the social and behavioral sciences (Pigott et al., 2013; Tanner-Smith & Polanin, 
2014). Given the increased awareness of publication bias among primary study authors, meta- 
analysts, peer reviewers, and journal editors, we may find less evidence of publication bias going 
forward. Any such improvements henceforth, however, will not counteract the existence of 
publication bias in decades past. Systematic reviewers and meta-analysts must continue to 
search for, and include, grey literature in their reviews because of this phenomenon. Peer 
reviewers and editors should ask for, at a minimum, a clear description of the study selection and 
sources of literature. At most, this important stakeholder group could demand the inclusion of 
grey literature. Meta-analytic consumers must be aware of the literature body synthesized. The 
consumer should view reviews of published literature with a level of skepticism until a review of 
the entire body of literature is completed. 

The results of this meta-review indicated that a failure to include unpublished studies in 
an education or psychology meta-analysis likely yields a biased overall average effect size 
estimate. Moreover, simply including one grey literature source, such as dissertations, is not a 
suitable solution to abate potential publication bias. We hope this research inspires future meta- 
analysts to consider the myriad forms of publications available and motivates methodologists to 
investigate this paradigm in the social sciences. For meta-analyses to remain relevant and 
valuable, review authors simply must include all available literature, regardless of publication 


status. 
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Table 1 

Included Meta-Analyses ’ Characteristics 


First Author Date Source Synthesis Description Outcome; Effect Funded; Grey Total Studies; 


Size; Model Bias Literature Unpublished 


Astill 

2012 

PB 

Relationships between sleep, cognitive 
performance, and behavioral problems 

P, A; C; R 

Y; T, 

FP 

D; Co 

86; 17 

B angert-Drowns 

1991 

RER 

Intervention effects of feedback on 
achievement 

A; S; N 

N; M 

D; G 

58; 28 

B angert-Drowns 

1993 

RER 

Intervention effects of word processors on 
achievement 

A; S; N 

N; M 

D 

20; 14 

B angert-Drowns 

2004 

RER 

Intervention effects of write-to-learn programs 
on achievement 

A; S; F 

Y; M 

G 

48; 13 

Bosch 

2006 

PB 

Difference between human and number 
generators 

P; S; B 

Y; T 

G 

355; 78 

Bowman 

2011 

RER 

Correlation between college diversity and civic 
engagement 

P; C; R 

N; T 

D 

28; 5 

Byrnes 

1999 

PB 

Gender differences in risk taking 

P; S; N 

N; M 

D 

150; 15 

Cantor 

2005 

PB 

Difference between sexual offenders and non- 
offenders on IQ 

P; S; R 

Y; M 

D; G 

161; 22 

Cooper 

2003 

RER 

Intervention effects of calendar modifications 

A; S; B 

Y; M 

D; Co; G 

41; 33 

Cooper 

2006 

RER 

Relationship between time on homework and 
academic achievement 

P, A; C; F 

Y; T 

D; Co; G 

35; 18 

Cooper 

2010 

RER 

Intervention effects of full day vs half day 
kindergarten 

A; S; B 

N; T 

D; G 

40; 33 

Cornelius-White 

2007 

RER 

Relationship between teacher-student dyad and 
achievement 

A; C; R 

Y; M 

D; BC; 

B; C; G 

118; 40 

Currier 

2008 

PB 

Intervention effects of psychotherapy on 
bereavement 

P; S; R 

N; T 

D 

64; 16 

Deci 

1999 

PB 

Experimental effects of rewards on intrinsic 
motivation 

P; S; N 

Y; M 

D 

70; 21 

Dindia 

1992 

PB 

Sex differences in self disclosure 

P; S; N 

N; M 

D 

200; 51 

Douglas 

2009 

PB 

Relationship between psychosis and violence 

P; C; R 

N; F 

D 

166; 6 

Dush 

1989 

PB 

Intervention effects of self-statement 
modification on child behavior disorders 

P; S; N 

N; M 

D; Co 

48; 16 



PUBLISHED AND UNPUBLISHED EFFECT SIZES 


41 


Eagly 

1986 

PB 

Sex differences in aggressive behavior 

P; S; N 

Y; M 

D 

50; 8 

Eagly 

1991 

PB 

Physical attractiveness differences on life 
outcomes 

P; S; N 

Y; M 

D; G 

76; 23 

Eagly 

1999 

PB 

Experimental effects of attitudes on memories 

P; S; F 

Y; M 

D; Co; G 

62; 18 

Eagly 

2003 

PB 

Women vs men in leadership style 

P; S; R 

Y; M 

D; B; G 

44; 28 

Elfenbein 

2002 

PB 

Relationship between cultural specificity and 
emotion recognition 

P; C; N 

Y; FD 

D; Co; B; 
BC; G 

43; 17 

Erel 

1995 

PB 

Marital relations and parent-child relationship 

P; C; N 

Y; M 

D; BC 

69; 22 

Feingold 

1988 

PB 

Relationship between partner attractiveness: 
samples 

P; C; N 

N; M 

D; Co; B 

19; 6 

Frattaroli 

2006 

PB 

Experimental effects of disclosure 

P; S; B 

Y; FD 

D; Co; G 

146; 70 

Glasman 

2006 

PB 

Relationship between attitudes and future 
behavior 

P; C; B 

Y; M 

D; G 

42; 8 

Gliessman 

1988 

RER 

Intervention effect of teaching skills on 
achievement 

A; S; F 

N; M 

G 

26; 17 

Grabe 

2006 

PB 

Difference between races on body 
dissatisfaction 

P; S; R 

N; M 

D 

93; 16 

Grabe 

2008 

PB 

Experimental effects of media on body image 

P; S; R 

Y; M 

D; G 

20; 3 

Hattie 

1996 

RER 

Whether certain interventions improve 
academic performance, study skills, or affect 

A; S; N 

Y; M 

D; BC; G 

51; 6 

Hostetter 

2011 

PB 

Gestures vs no gestures in communication 

A; S; R 

N; F 

D; Co 

63; 15 

Jaffee 

2000 

PB 

Sex differences in moral orientation 

P; S; N 

N; M 

D; Co; B; 

n 

114; 44 

Johnson 

2006 

PB 

Experimental effect: personal goal vs 
accessibility of goal-related attainment 

A; S; F 

N; T 

yj 

D; Co 

27.5; 7 

Judge 

2001 

RER 

Correlation between job satisfaction and 
performance 

P; C; N 

N; M 

D 

312; 92 

Juffer 

2007 

PB 

Difference between adopt vs non-adopt 

P; S; R 

Y; T 

B; G 

88; 44 

Kim 

2013 

RER 

Effect of summer reading on reading 
comprehension 

A; S; R 

N; T, 

FD 

D; BC; G 

35; 24 

Koenig 

2011 

PB 

Relationship between leadership stereotypes 
and masculinity 

P; C; R 

Y; T, 

FP 

D; G 

50; 12 

Kulik 

1988 

RER 

Intervention effects of immediate feedback 

A; S; N 

N; M 

D 

27; 3 

LaFrance 

2003 

PB 

Difference of sexes 

P; S; F 

Y;F 

D; B; 

BC; Co 

418; 126 

Lanaj 

2012 

PB 

Antecedents and consequences of regulatory 

P; C; R 

N; M 

D; Co 

125; 55 
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focus 


Lauer 

2006 

RER 

Intervention effects of out of school programs 

A; S; R 

Y; M 

D; Co; G 

26; 22 

Lou 

1996 

RER 

Intervention effects of grouping on academic 
achievement 

A; S; N 

Y; M 

D; Co; G 

105; 17 

Lou 

2001 

RER 

Intervention effects of grouping on computer 
technology learning 

A; S; N 

Y; M 

D; Co; G 

100; 48 

Lytton 

1991 

PB 

Parental differential towards boys and girls 

P; S; N 

Y; M 

D; Co; G 

40; 8 

Marulis 

2010 

RER 

Intervention effects of vocab interventions 

A; S; R 

Y;F 

D; G 

62; 15 

McClure 

2000 

PB 

Sex differences in facial expression processing 

P; S; R 

N; FD 

D 

80; 21 

Metzger 

2008 

RER 

Relationship between teacher perceiver 
interview and teacher quality 

A; C; B 

Y; FP 

D; G 

45; 43 

Mor 

2002 

PB 

Relationship between attention and negative 
affect 

P; C; N 

N; F 

D; G 

104; 20 

Murray 

2012 

PB 

Difference between children with and without 
incarcerated parents 

P, A; C; R 

Y; T 

D 

40; 20 

Nesbit 

2006 

RER 

If using concept maps increased knowledge 
retention 

A; S; R 

Y; M 

D 

40; 12 

Pascoe 

2009 

PB 

Relationship between discrimination and health 

P; C; B 

Y; T 

D; Co; G 

192; 19 

Patall 

2008 

RER 

Relationship between parental involvement and 
achievement 

A; C; B 

N; T 

D; Co; G 

22; 11 

Patall 

2008 

PB 

Experimental effects of choice on intrinsic 
motivation 

P; S; B 

N; T 

D 

46; 18 

Penny 

2004 

PB 

Intervention effects of consultation on teaching 
effectiveness 

A; S; F 

N; M 

D 

11; 4 

Qin 

1995 

RER 

Intervention effect 

A; S; N 

N; M 

D; B 

59; 10 

Rhodes 

2011 

PB 

Experimental effects of delayed judgment on 
metacognitive accuracy 

A; S; F 

N; FD 

D; G 

112; 22 

Rhodes 

2012 

PB 

Facial recognition of one's own age vs. other 
ages 

P; S; R 

N; FD 

Co; G; 

BC 

46; 4 

Rind 

1998 

PB 

Adjustment of victims of child sexual abuse 

P; C; N 

N; M 

D 

62; 23 

Ritter 

2009 

RER 

Intervention effects of volunteer tutor programs 

A; S; R 

Y; T 

D; G 

14; 5 

Ross 

1988 

RER 

Intervention effects of programs on students 

A; S; N 

N; M 

D; G 

61; 20 

Sedlmeier 

2012 

PB 

Intervention effect of meditation on 
emotionality and attention 

P; S; F 

N; FD, 
FP 

D; BC 

164; 39 

Seto 

2010 

PB 

Difference between male sex offenders and 
others 

P; S; R 

N; M 

D; BC; 

Co; G 

28; 17 
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Smith 

2003 

PB 

Relationship between religiousness and 
depression 

P; C; R 

Y; FD, 
FP 

D; Co 

147; 36 

Sosa 

2011 

RER 

Intervention effects of computer-assisted 
learning 

A; S; B 

N; FD 

D; Co 

45; 13 

Sowislo 

2013 

PB 

Relationship between self-esteem and 
depression 

P; C; R 

Y; FP 

D 

77; 7 

Steblay 

1987 

PB 

Relationship between helping behavior and 
living environment 

P; C; N 

Y; FD 

D; Co 

35; 4 

Stice 

2006 

PB 

Intervention effects of prevention programs on 
obesity 

P; S; R 

Y; M 

D 

64; 2 

Swanson 

1998 

RER 

Intervention effects of programs for learning 
disabled students 

A; S; F 

Y;F 

D 

180; 25 

Thoresen 

2008 

RER 

Correlation between job attitude and 
performances 

P; C; R 

N; F 

D 

37; 7 

Toosi 

2012 

PB 

Interracial interactions and attitudes, emotional 
states 

P, A; C; R 

Y; T, 

FP 

D; G 

81; 9 

Uttal 

2013 

PB 

Intervention effects on spatial skills 

A; S; R 

Y; FD, 
FP 

D; BC; 

B; Co; G 

206; 111 

Vachon 

2013 

PB 

Relationship between empathy and aggression 

P; C; B 

N; FP 

D 

103; 31 

Van den Bussche 

2009 

PB 

Before vs after reaction times for different 
stimulus 

P; S; R 

N; T 

D; G 

28; 4 

Van Ijzendoorn 

2005 

PB 

Difference between adopted and non-adopted 
kids 

A; S; R 

Y; T 

D; B; 

BC; G 

36; 4 

Veenman 

1996 

RER 

Multi-grade vs single-grade class effects 

A; S; R 

N; M 

D; B; G 

40; 29 

Vitaliano 

2003 

PB 

Difference between caregivers and non- 
caregivers on physical health 

P; S; R 

Y; T 

D 

17; 2 

Webb 

2006 

PB 

Intervention effect on intention-change 

A; S; R 

N; M 

D; G 

47; 5 

Weisz 

2006 

PB 

Intervention effects of psychotherapy on 
depression 

P; S; R 

Y; M 

D 

35; 8 

Wood 

1991 

PB 

Experimental effects of media violence on 
aggression 

P; S; N 

Y; M 

D; Co 

14; 4 

Wood 

1994 

PB 

Experimental effects of minority influence on 
social processes 

P; S; N 

Y; M 

D 

23; 6 

Wood 

2003 

PB 

Experimental effects of forewarning on self- 
image 

P; S; R 

Y; M 

D 

20; 2 
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Note. First author represents the meta-analyses’ first author’s last name; RER = Review of Educational Research ; PB = Psychological Bulletin ; A = Achievement 
outcome; P = Psychological outcome; S = Standardized mean-difference; C = Correlation coefficient; N = Not reported; F = Fixed-effect; R = Random-effects; B 
= Both fixed- and random-effects; Y = Funded; M = Moderator only; T = Trim and fill; FD = File-drawer; FP = Funnel plot; D = Dissertation; G = General 
unpublished; B = Book; BC = Book chapter; Co = Conference proceeding. 
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Table 2 


Grey Literature Included in the Sample of Meta-Analyses 


Publication Types of Studies 
included in the Meta- Analysis 

Meta-Analyses 
with Pub. 
Type (%) 

Number of 
Studies 
Included in 
Meta-Analysis 
(Mean, SD) 

Minimum, 

Maximum 

Average % 
of Total 

Journal Articles 

81 (100) 

55.74 (56.74) 

2, 292 

68.5 

Unpublished Literature 

81 (100) 

22.19 (23.11) 

2, 126 

31.5 

Dissertations/Theses 

64 (79.0) 

13.43 (18.42) 

0, 92 

18.0 

Conference Abstracts 

26 (32.1) 

2.34 (8.20) 

0, 68 

2.80 

Books ( General ) 

8 (9.87) 

0.56 (2.96) 

0, 25 

0.52 

Book Chapters 

12(14.8) 

0.82 (3.40) 

0,9 

0.81 

General Unpublished 

45 (55.6) 

5.03 (2.96) 

0, 45 

9.40 


Note, k = 81; Unpublished literature row represents the totality of unpublished studies; General 


unpublished is used when review authors did not provide grey literature types or to represent all 
other types of unpublished literature (i.e., unpublished datasets, reports, etc.). 
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Table 3 


Meta-Regression Analysis 


Model 

I 

II 

III 

IV 

V 

VI 

VII 

Characteristics 








Effect size type 

-0.01 






-0.14 


[-0.83,0.81] 






[-0.96, 0.68] 

Dissertations only 


-0.77* 





-0.83* 



[-1.51,-0.03] 





[-1.63,-0.03] 

Percent unpublished 



-0.03 




-0.19 




[-1.57, 1.51] 




[-.1.83, 1.46] 

Funded 




0.27 



0.18 





[-0.42, 0.95] 



[-0.49, 0.85] 

Date of publication 





-0.03 


-0.04 






[-0.08, 0.01] 


[-0.09, 0.01] 

Q-between conversion 






0.47 

0.60 







[-0.42, 1.36] 

[-0.34, 1.53] 

Intercept 

0.80* 

1.03* 

0.81* 

0.65* 

0.75* 

0.71* 

0.94* 


[0.06, 1.54] 

[0.60, 1.45] 

[0.18, 1.44] 

[0.26, 1.05] 

[0.42, 1.08] 

[0.33, 1.10] 

[0.13,2.01] 

R 2 

5.38 

15.16 

1.97 

11.01 

0.01 

0.01 

5.16 


Note. Number of effect sizes = 147; Number of meta-analyses = 81; Results estimated from mixed-effect meta-regression models with 
robust standard errors; Numbers in brackets represent 95% confidence intervals; Effect size type (0 = Correlation coefficient, 1 = 
Standardized mean-difference); Dissertations only (1 = Dissertations only); Funded (1 = Received funding); Q-Between conversion (1 
= Converted from <2-between tests); * p < .05. 
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Publication Source Moderator Analyses (Total) 
Meta-analyses (Total) 


Figure 1. Number of Moderator Analyses Conducted Per Year. 
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Online Supplemental Figure SI: Forest Plot of all Effect Sizes Calculated. 


